Skip to content

feat(hub): add optional OpenTelemetry server tracing - #36

Open
Copilot wants to merge 6 commits into
mainfrom
copilot/hub-opentelemetry-tracing
Open

Copilot wants to merge 6 commits into
mainfrom
copilot/hub-opentelemetry-tracing

Conversation

Copilot AI commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

Companies running their own Hub need to connect their application performance monitoring (APM) provider without rebuilding the published container image. A shared OpenTelemetry integration lets operators choose a provider instead of requiring Hub to bundle each vendor's SDK.

Closes #34.

What changes

Hub can export server-side traces through the OpenTelemetry Protocol (OTLP) over HTTP/protobuf. Tracing is off by default. Operators enable it with environment variables and restart every Hub web and job container. Traces cover Rails requests, background jobs, SQLite queries, and outbound Net::HTTP calls.

By default, Hub exports operational metadata without SQL text, job arguments, or exception messages and stack traces. REPLICA_OTEL_INCLUDE_SENSITIVE_DATA=true enables those diagnostics together. Credential-keyed job fields remain filtered, SQL bind values remain excluded, and request headers and bodies are never attached.

OpenTelemetry database spans with sensitive diagnostics disabled and enabled

Public requests start fresh traces unless the operator enables REPLICA_OTEL_TRUST_INCOMING_CONTEXT behind an ingress that sanitizes trace headers. Hub discards incoming vendor trace state and baggage while preserving its own job trace continuity. Operators can exclude exact outbound hosts from both HTTP tracing and trace-header propagation.

Export runs in a bounded background queue. Collector or provider outages can drop traces but do not block successful requests. The deployment guide includes a copy-paste local Aspire dashboard setup and explains endpoint authentication, sampling, certificates, restart behavior, and the data impact of each Replica tracing setting.

Operators can send traces directly to a compatible provider or through a company-managed OpenTelemetry Collector. This first version does not add browser monitoring, metric or log export, an admin configuration page, or vendor-native error reporting. Live Sentry and Datadog ingestion has not been verified.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI max review requested due to automatic review settings October 19, 2026 19:02
Copilot AI requested review from a team and nshki as code owners October 19, 2026 19:02

This comment was marked as duplicate.

Comment thread hub/docs/deployment.md Outdated
Comment thread hub/lib/hub_tracing.rb Outdated
Copilot AI and others added 5 commits October 19, 2026 23:40
Include filtered job arguments, SQL text, and exception diagnostics, with an operator-controlled identity opt-in. Align exported metadata with bundled instrumentation and isolate public incoming trace context by default.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document identity opt-in, trusted ingress, outbound exclusions, and the limits of key-based filtering.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Centralize the sensitive-data policy in the exporter while avoiding collection of SQL, job arguments, and exception diagnostics unless operators explicitly opt in.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add a local Aspire dashboard workflow and show exactly which trace fields change when sensitive diagnostics are enabled.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread hub/lib/hub_tracing.rb
begin
uri = URI.parse(endpoint.to_s)
rescue URI::InvalidURIError
raise ArgumentError, "OTLP endpoint must be an absolute HTTP(S) URL"
Comment thread hub/lib/hub_tracing.rb
config.use "OpenTelemetry::Instrumentation::ActionPack", span_naming: :class
config.use "OpenTelemetry::Instrumentation::ActiveJob",
propagation_style: :child, span_naming: :job_class
config.use "OpenTelemetry::Instrumentation::Net::HTTP", untraced_hosts:
Comment thread hub/lib/hub_tracing.rb
next unless span.recording?

begin
span.set_attribute("replica.job.arguments", data_filter.job_arguments(job.arguments))
Comment thread hub/lib/hub_tracing.rb
# ORM instrumentation does not cover Hub's direct SQLite calls.
tracer = OpenTelemetry.tracer_provider.tracer("replica-hub")
ActiveSupport::Notifications.subscribe("sql.active_record") do |_, started, finished, _, payload|
next if payload[:name] == "SCHEMA" || payload[:cached]
Comment on lines +124 to +126
assert spans.any? do |span|
span_attributes(span)["db.query.text"]&.string_value&.include?("sql-diagnostic-canary")
end
queued = TracingProbeJob.queue_adapter.enqueued_jobs.shift.select { |key, _| key.is_a?(String) }
serialized = JSON.parse(JSON.generate(queued))
begin
ActiveJob::Base.execute(serialized)
Comment thread hub/docs/deployment.md
| Error status | Exported without a description | Exported without a description |
| Exception type, message and stack trace | Excluded | Exported |

The exporter enforces this policy after instrumentation collects spans. Hub also avoids collecting SQL text, job arguments and exception events when the setting is `false`. Changing the environment variable in a running process does not change its policy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(hub): support optional OpenTelemetry server tracing

4 participants